Image generating apparatus, image generating system, image generating method, and program

ABSTRACT

An image generating apparatus includes a rendering unit carrying out rendering of an object of a virtual space and an object of a real space and carrying out rendering of expression relating to light of the virtual space with respect to the real space to generate a computer graphics image, a superimposing unit superimposing the computer graphics image on a photographed image of the real space to generate a provisional superposition image, a chroma key generating unit generating a chroma key image through executing chroma key processing for the computer graphics image based on depth information of the photographed image of the real space, and a synthesizing unit generating a synthesized chroma key image used for being superimposed on the photographed image of the real space to generate an augmented reality image by applying a mask to the provisional superposition image by the chroma key image.

BACKGROUND

This disclosure relates to an apparatus, a system, and a method thatgenerate an image.

Mounting a head-mounted display connected to a game machine on a headand playing a game through operation of a controller or the like whileviewing a screen displayed on the head-mounted display have been carriedout. When the head-mounted display is mounted, there is an effect that asense of immersion in a video world is enhanced and the entertainmentproperty of the game is further enhanced because the user views nothingother than the video displayed on the head-mounted display. Furthermore,if video of virtual reality (VR) is displayed on the head-mounteddisplay and an omnidirectional virtual space in which a 360-degree viewcan be seen when the user who wears the head-mounted display rotates thehead is allowed to be displayed, the sense of immersion in the video isfurther enhanced and the operability of the application of a game or thelike is also improved.

Furthermore, although a user who wears a non-transmissive head-mounteddisplay becomes incapable of directly seeing the external world, thereis also a head-mounted display of a video-transmissive (videosee-through) type that can photograph video of the external world by acamera mounted on the head-mounted display and display the video on adisplay panel. In the head-mounted display of the video-transmissivetype, it is also possible to generate and display video of augmentedreality (AR) by superimposing objects of a virtual world generated basedon computer graphics (CG) on the video of the external worldphotographed by the camera. Differently from the virtual realityseparated from the real world, the video of the augmented reality iswhat is obtained through augmentation of the real world by the virtualobjects, and the user can experience the virtual world while being awareof a connection with the real world.

SUMMARY

In the case of superimposing a virtual object generated based on CG on acamera image to generate video of augmented reality and display thevideo on a head-mounted display, aliasing occurs at the boundary of thevirtual object due to the influence of a post-process on the image andthe boundary between the virtual world and the real world is conspicuousand AR video with a sense of unity is not obtained in some cases.Furthermore, if a shadow cast on the real space by a virtual object anda reflection of the virtual object onto the real space are not reflectedin the AR video, a sense of unity between the virtual world and the realworld is not obtained and the virtual object looks like being out ofplace in the real world.

The present disclosure is made in view of such problems and there is aneed for providing an image generating apparatus, an image generatingsystem, and an image generating method that can improve the quality ofvideo of augmented reality.

According to an embodiment of the present disclosure, there is providedan image generating apparatus including a rendering unit configured tocarry out rendering of an object of a virtual space and an object of areal space and carry out rendering of expression relating to light ofthe virtual space with respect to the real space to generate a computergraphics image, a superimposing unit configured to superimpose thecomputer graphics image on a photographed image of the real space togenerate a provisional superposition image, a chroma key generating unitconfigured to generate a chroma key image through executing chroma keyprocessing for the computer graphics image based on depth information ofthe photographed image of the real space, and a synthesizing unitconfigured to generate a synthesized chroma key image used for beingsuperimposed on the photographed image of the real space to generate anaugmented reality image by applying a mask to the provisionalsuperposition image by the chroma key image. The chroma key generatingunit employs a region of the real space in which the object of thevirtual space is not rendered as a chroma key region and does not employa region of the real space in which the expression relating to the lightof the virtual space exists as a chroma key region.

According to another embodiment of the present disclosure, there isprovided an image generating system including a head-mounted display andan image generating apparatus. The image generating apparatus includes arendering unit configured to carry out rendering of an object of avirtual space and an object of a real space and carry out rendering ofexpression relating to light of the virtual space with respect to thereal space to generate a computer graphics image, a first superimposingunit configured to superimpose the computer graphics image on aphotographed image of the real space transmitted from the head-mounteddisplay to generate a provisional superposition image, a chroma keygenerating unit configured to generate a chroma key image throughexecuting chroma key processing for the computer graphics image based ondepth information of the photographed image of the real spacetransmitted from the head-mounted display, and a synthesizing unitconfigured to generate a synthesized chroma key image used for beingsuperimposed on the photographed image of the real space to generate anaugmented reality image by applying a mask to the provisionalsuperposition image by the chroma key image. The head-mounted displayincludes a second superimposing unit configured to generate theaugmented reality image by synthesizing the photographed image of thereal space with the synthesized chroma key image transmitted from theimage generating apparatus. The chroma key generating unit employs aregion of the real space in which the object of the virtual space is notrendered as a chroma key region and does not employ a region of the realspace in which the expression relating to the light of the virtual spaceexists as a chroma key region.

According to a further embodiment of the present disclosure, there isprovided an image generating method including carrying out rendering ofan object of a virtual space and an object of a real space and carryingout rendering of expression relating to light of the virtual space withrespect to the real space to generate a computer graphics image,superimposing the computer graphics image on a photographed image of thereal space to generate a provisional superposition image, generating achroma key image through executing chroma key processing for thecomputer graphics image based on depth information of the photographedimage of the real space, and generating a synthesized chroma key imageused for being superimposed on the photographed image of the real spaceto generate an augmented reality image by applying a mask to theprovisional superposition image by the chroma key image. The generatinga chroma key image employs a region of the real space in which theobject of the virtual space is not rendered as a chroma key region anddoes not employ a region of the real space in which the expressionrelating to the light of the virtual space exists as a chroma keyregion.

What are obtained by translating arbitrary combinations of the aboveconstituent elements and expressions of the present disclosure amongmethod, apparatus, system, computer program, data structure, recordingmedium, and so forth are also effective as embodiments of the presentdisclosure.

According to the present disclosure, the quality of video of augmentedreality can be improved.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an appearance diagram of a head-mounted display;

FIG. 2 is a configuration diagram of an image generating systemaccording to embodiments of the present disclosure;

FIG. 3 is a diagram explaining an example of a camera image photographedby a camera mounted on the head-mounted display of FIG. 1;

FIG. 4 is a diagram explaining an augmented reality image obtained bysuperimposing a virtual object based on CG on the camera image of FIG.3;

FIG. 5 is a diagram illustrating a state in which a user reaches out ahand for the virtual object with respect to the augmented reality imageof FIG. 4;

FIG. 6 is a diagram explaining a CG image used for chroma key synthesis;

FIG. 7 is a functional configuration diagram of a head-mounted displayaccording to a premise technique;

FIG. 8 is a functional configuration diagram of an image generatingapparatus according to the premise technique;

FIG. 9 is a diagram explaining the configuration of an image generatingsystem according to the premise technique for superimposing a CG imageon a camera image to generate an augmented reality image;

FIG. 10 is a functional configuration diagram of an image generatingapparatus according to a first embodiment of the present disclosure;

FIG. 11 is a diagram explaining the configuration of an image generatingsystem according to the first embodiment for superimposing a CG image ona camera image to generate an augmented reality image;

FIG. 12 is a diagram explaining an augmented reality image obtained bysuperimposing a CG image on a camera image by the image generatingsystem according to the first embodiment;

FIG. 13 is a diagram explaining a synthesized CG chroma key image usedby the image generating system according to the first embodiment;

FIG. 14 is a diagram explaining an example in which a polygon mesh of areal space is deformed to make a hole in a wall;

FIG. 15 is a diagram explaining an example in which a virtual object isrendered in the hole of the wall of the real space;

FIG. 16 is a functional configuration diagram of an image generatingapparatus according to a second embodiment of the present disclosure;and

FIG. 17 is a diagram explaining the configuration of an image generatingsystem according to the second embodiment for superimposing a CG imageon a camera image to generate an augmented reality image.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

FIG. 1 is an appearance diagram of a head-mounted display 100. Thehead-mounted display 100 is a display apparatus for being mounted on thehead of a user to allow the user to view a still image, moving image, orthe like displayed on a display and listen to sound, music, or the likeoutput from a headphone.

Position information of the head of the user who wears the head-mounteddisplay 100 and orientation information such as the rotational angle andtilt of the head can be measured by gyro sensor, acceleration sensor,and so forth incorporated in or externally attached to the head-mounteddisplay 100.

The head-mounted display 100 is equipped with a camera unit and theexternal world can be photographed while the user wears the head-mounteddisplay 100.

The head-mounted display 100 is one example of a “wearable display.”Here, a generating method of an image displayed on the head-mounteddisplay 100 will be described. However, the image generating method ofthe present embodiment can be applied to not only the case in which auser wears the head-mounted display 100 in a narrow sense but also thecase in which a user wears eyeglasses, eyeglasses-type display,eyeglasses-type camera, headphone, headset (headphone equipped with amicrophone), earphone, earring, ear-hook camera, headwear, headwearequipped with a camera, hair band, or the like.

FIG. 2 is a configuration diagram of an image generating systemaccording to the present embodiment. As one example, the head-mounteddisplay 100 is connected to an image generating apparatus 200 by aninterface 300 of the high-definition multimedia interface (HDMI)(registered trademark), which is a standard specification of acommunication interface to transmit video and sound by a digital signal,or the like.

The image generating apparatus 200 predicts position-orientationinformation of the head-mounted display 100 in consideration of delayfrom generation of video to display thereof from the presentposition-orientation information of the head-mounted display 100 andrenders the image to be displayed on the head-mounted display 100 on thepremise of the predicted position-orientation information of thehead-mounted display 100 to transmit the image to the head-mounteddisplay 100.

One example of the image generating apparatus 200 is a game machine. Theimage generating apparatus 200 may be further connected to a serverthrough a network. In this case, the server may provide an onlineapplication such as a game in which plural users can participate throughthe network to the image generating apparatus 200. The head-mounteddisplay 100 may be connected to a computer or portable terminal insteadof the image generating apparatus 200.

With reference to FIG. 3 to FIG. 6, an augmented reality image obtainedby superimposing a virtual object based on CG on a camera image will bedescribed.

FIG. 3 is a diagram explaining an example of a camera image photographedby the camera mounted on the head-mounted display 100. This camera imageis what is obtained by photographing a table and a basket 400 existingthereon, with a room being the background. The surface of the table isgiven a floral pattern. Although the background hardly changes in thecamera image, a user reaches out a hand and moves the basket 400existing on the table in some cases.

FIG. 4 is a diagram explaining an augmented reality image obtained bysuperimposing a virtual object based on CG on the camera image of FIG.3. The basket 400, which is a real object existing on the table, isreplaced by a teapot 410 that is the virtual object generated based onCG and the teapot 410 is superimposed on the camera image. This allowsthe user to view the augmented reality image in which the virtual objectis drawn in the real space by the head-mounted display 100.

FIG. 5 is a diagram illustrating a state in which the user reaches out ahand for the virtual object with respect to the augmented reality imageof FIG. 4. When the user who is viewing the augmented reality image bythe head-mounted display 100 attempts to touch the teapot 410, which isthe virtual object, the hand of the user is photographed by the cameramounted on the head-mounted display 100 and thus a hand 420 is capturedin the camera image. The teapot 410, which is the virtual object, issuperimposed on the camera image in which the hand 420 is captured. Atthis time, it is desired to correctly determine the positional relationbetween the teapot 410 and the hand 420 by using depth information suchthat the obtained image may be kept from becoming an unnatural augmentedreality image such as an image in which the teapot 410 is superimposedon the hand 420 and the hand 420 becomes invisible.

Therefore, the depth information of the camera image is used todetermine the positional relation between a thing captured in the cameraimage and the virtual object and rendering in which the depth iscorrectly reflected is carried out. The depth is known in advanceregarding the background of the room and the basket 400 regarding whichexistence has been already known, and therefore the positional relationwith the virtual object can be determined in advance. However, when theuser reaches out a hand or foot or in the case in which a moving bodyother than the user (another person, dog, cat, or the like, for example)comes into the field of view, or the like, the depth is not known inadvance and therefore it is desired to determine the depth from thedepth information of the camera image on each occasion.

In general, when a CG image is superimposed on a camera image, a chromakey image obtained by painting out, with specific one color, the regionthat is not rendered, such as the background, in the CG image is createdand is used for chroma key synthesis. The region of the color specifiedas the chroma key (referred to as “chroma key region”) becomestransparent. Therefore, when the chroma key image is superimposed on thecamera image, the camera image is displayed in the chroma key region.

FIG. 6 is a diagram explaining the CG image used for the chroma keysynthesis. In the state of FIG. 5, the background is pained out with thespecific color of the chroma key (red, for example). Furthermore, thehand 420 captured in the camera image exists on the near side relativeto the teapot 410. Thus, the region hidden by the hand 420 in the regionof the teapot 410 is also painted out with the specific color of thechroma key. When this chroma key image is superimposed on the cameraimage, the camera image is left and the augmented reality image of FIG.5 is obtained because the part of the specific color of the chroma keyis transparent.

FIG. 7 is a functional configuration diagram of the head-mounted display100 according to a premise technique.

A control unit 10 is a main processor that processes and outputs signalssuch as image signal and sensor signal and instructions and data. Aninput interface 20 accepts operation signal and setting signal from auser and supplies them to the control unit 10. An output interface 30receives the image signal from the control unit 10 and displays an imageon a display panel 32.

A communication control unit 40 transmits data input from the controlunit 10 to the external through a network adapter 42 or an antenna 44 bywired or wireless communication. Furthermore, the communication controlunit 40 receives data from the external and outputs the data to thecontrol unit 10 through the network adapter 42 or the antenna 44 bywired or wireless communication.

A storing unit 50 temporarily stores data, parameters, operation signal,and so forth processed by the control unit 10.

An orientation sensor 64 detects position information of thehead-mounted display 100 and orientation information such as therotational angle and tilt of the head-mounted display 100. Theorientation sensor 64 is implemented by appropriately combining gyrosensor, acceleration sensor, angular acceleration sensor, and so forth.The forward-rearward, rightward-leftward, upward-downward motion of thehead of the user may be detected by using a motion sensor obtained bycombining at least one or more of three-axis geomagnetic sensor,three-axis acceleration sensor, and three-axis gyro (angular velocity)sensor.

An external input-output terminal interface 70 is an interface forconnecting peripheral equipment such as a universal serial bus (USB)controller. An external memory 72 is an external memory such as a flashmemory.

A camera unit 80 includes configurations for photographing, such aslens, image sensor, and ranging sensor, and supplies video and depthinformation of the photographed external world to the control unit 10.The control unit 10 controls focus, zoom, and so forth of the cameraunit 80.

An image signal processing unit 82 executes image signal processing(ISP) such as RGB conversion (demosaic processing), white balance, colorcorrection, and noise reduction for a Raw image photographed by thecamera unit 80, and executes distortion correction processing ofremoving distortion and so forth attributed to the optical system of thecamera unit 80. The image signal processing unit 82 supplies a cameraimage for which the image signal processing and the distortioncorrection processing have been executed to the control unit 10.

A reprojection unit 84 executes reprojection processing for the cameraimage based on the latest position-orientation information of thehead-mounted display 100 detected by the orientation sensor 64 toconvert the image to an image viewed from the latest position of thepoint of view and the latest direction of the line of sight of thehead-mounted display 100.

A distortion processing unit 86 executes processing of deforming anddistorting an image in conformity to distortion that occurs in theoptical system of the head-mounted display 100 for the camera image forwhich the reprojection processing has been executed, and supplies thecamera image for which the distortion processing has been executed tothe control unit 10.

An AR superimposing unit 88 generates an augmented reality image bysuperimposing a CG image generated by the image generating apparatus 200on the camera image for which the distortion processing has beenexecuted, and supplies the augmented reality image to the control unit10.

An HDMI transmitting-receiving unit 90 transmits and receives a digitalsignal of video and sound to and from the image generating apparatus 200in accordance with the HDMI. The HDMI transmitting-receiving unit 90receives, from the control unit 10, the RGB image for which the imagesignal processing and the distortion correction processing have beenexecuted by the image signal processing unit 82 and depth informationand transmits them to the image generating apparatus 200 by an HDMItransmission path. The HDMI transmitting-receiving unit 90 receives animage generated by the image generating apparatus 200 from the imagegenerating apparatus 200 by the HDMI transmission path and supplies theimage to the control unit 10.

The control unit 10 can supply an image or text data to the outputinterface 30 to cause the display panel 32 to display it, and supply itto the communication control unit 40 to cause the communication controlunit 40 to transmit it to the external.

The present position-orientation information of the head-mounted display100 detected by the orientation sensor 64 is notified to the imagegenerating apparatus 200 through the communication control unit 40 orthe external input-output terminal interface 70. Alternatively, the HDMItransmitting-receiving unit 90 may transmit the presentposition-orientation information of the head-mounted display 100 to theimage generating apparatus 200.

FIG. 8 is a functional configuration diagram of the image generatingapparatus 200 according to the premise technique. This diagram depicts ablock diagram in which attention is paid to functions and thesefunctional blocks can be implemented in various forms by only hardware,only software, or a combination of them.

At least part of the functions of the image generating apparatus 200 maybe implemented in the head-mounted display 100. Alternatively, at leastpart of the functions of the image generating apparatus 200 may beimplemented in a server connected to the image generating apparatus 200through a network.

A position-orientation acquiring unit 210 acquires the presentposition-orientation information of the head-mounted display 100 fromthe head-mounted display 100.

A point-of-view/line-of-sight setting unit 220 sets the position of thepoint of view and the direction of the line of sight of a user by usingthe position-orientation information of the head-mounted display 100acquired by the position-orientation acquiring unit 210.

An HDMI transmitting-receiving unit 280 receives the depth informationof video of a real space photographed by the camera unit 80 from thehead-mounted display 100 and supplies the depth information to a depthacquiring unit 250.

An image generating unit 230 reads out data for generation of computergraphics from an image storing unit 260 and carries out rendering ofobjects of a virtual space to generate a CG image. Then, the imagegenerating unit 230 generates a chroma key image from the CG image basedon the depth information of the camera image of the real space providedfrom the depth acquiring unit 250 and outputs the chroma key image tothe image storing unit 260.

The image generating unit 230 includes a rendering unit 232, a chromakey generating unit 235, a post-process unit 236, a reprojection unit240, and a distortion processing unit 242.

The rendering unit 232 generates the CG image through rendering ofobjects of a virtual space that are visible in the direction of the lineof sight from the position of the point of view of the user who wearsthe head-mounted display 100 in accordance with the position of thepoint of view and the direction of the line of sight of the user set bythe point-of-view/line-of-sight setting unit 220, and gives the CG imageto the chroma key generating unit 235.

The chroma key generating unit 235 generates a chroma key image from theCG image based on the depth information of the camera image given fromthe depth acquiring unit 250. Specifically, the chroma key generatingunit 235 determines the positional relation between the objects of thereal space and the objects of the virtual space and generates the chromakey image (referred to as “CG chroma key image”) obtained by paintingout, with specific one color (red, for example), the background of thevirtual objects and the part of the object of the real space existing onthe near side relative to the virtual object in the CG image.

The post-process unit 236 executes a post-process such as adjustment ofthe depth of field, tone mapping, and anti-aliasing for the CG chromakey image and executes post-processing in such a manner that the CGchroma key image looks natural and smooth.

The reprojection unit 240 receives the latest position-orientationinformation of the head-mounted display 100 from theposition-orientation acquiring unit 210 and executes reprojectionprocessing for the CG chroma key image for which the post-process hasbeen executed to convert the image to an image viewed from the latestposition of the point of view and the latest direction of the line ofsight of the head-mounted display 100.

Here, a description will be made regarding the reprojection. In the caseof allowing the head-mounted display 100 to have a head-trackingfunction and generating video of virtual reality with change in thepoint of view and the direction of the line of sight in conjunction withthe motion of the head of the user, delay exists from the generation ofthe video of virtual reality to display thereof. Thus, a deviationoccurs between the direction of the head of the user employed as thepremise at the time of video generation and the direction of the head ofthe user at the timing when the video is displayed on the head-mounteddisplay 100 and the user falls into a sick-like feeling (called virtualreality sickness (VR sickness) or the like) in some cases.

As above, a long time is taken until the motion of the head-mounteddisplay 100 is detected and a central processing unit (CPU) issues arendering command and a graphics processing unit (GPU) executesrendering and a rendered image is output to the head-mounted display100. Suppose that the rendering is carried out at a frame rate of 60 fps(frame/seconds), for example, and delay corresponding to one frameoccurs from detection of the motion of the head-mounted display 100 tooutput of the image. This is approximately 16.67 milliseconds under theframe rate of 60 fps and is a sufficient time for the human to perceivethe deviation.

Therefore, processing called “time warp” or “reprojection” is executedand the rendered image is corrected in conformity to the latest positionand orientation of the head-mounted display 100 to cause the human toperceive the deviation less readily.

The distortion processing unit 242 executes processing of deforming anddistorting an image in conformity to distortion that occurs in theoptical system of the head-mounted display 100 for the CG chroma keyimage for which the reprojection processing has been executed, andstores the resulting image in the image storing unit 260.

The HDMI transmitting-receiving unit 280 reads out frame data of the CGchroma key image generated by the image generating unit 230 from theimage storing unit 260 and transmits the frame data to the head-mounteddisplay 100 in accordance with the HDMI.

FIG. 9 is a diagram explaining the configuration of an image generatingsystem according to the premise technique for superimposing a CG imageon a camera image to generate an augmented reality image. Here, forsimplification of the description, the main configuration of thehead-mounted display 100 and the image generating apparatus 200 forgenerating an augmented reality image is diagrammatically representedand will be described.

Camera image and depth information of the external world photographed bythe camera unit 80 of the head-mounted display 100 are supplied to theimage signal processing unit 82. The image signal processing unit 82executes the image signal processing and the distortion correctionprocessing for the camera image and gives the resulting image to thereprojection unit 84. The image signal processing unit 82 transmits thedepth information to the image generating apparatus 200 to supply it tothe chroma key generating unit 235.

The rendering unit 232 of the image generating apparatus 200 generatesvirtual objects viewed from the position of the point of view and thedirection of the line of sight of the user who wears the head-mounteddisplay 100 and gives the virtual objects to the chroma key generatingunit 235.

The chroma key generating unit 235 generates a CG chroma key image froma CG image based on the depth information. The post-process unit 236executes the post-process for the CG chroma key image. The reprojectionunit 240 converts the CG chroma key image for which the post-process hasbeen executed in conformity to the latest position of the point of viewand the latest direction of the line of sight. The distortion processingunit 242 executes the distortion processing for the CG chroma key imageafter the reprojection. The final RGB image after the distortionprocessing is transmitted to the head-mounted display 100 and issupplied to the AR superimposing unit 88. This RGB image is an image inwhich the region on which the camera image should be superimposed ispainted out with one color (red, for example) specified in the chromakey synthesis. The one color specified for the chroma key is not used asthe CG image. Therefore, expression is carried out by using anothercolor with avoidance of the one color specified for the chroma key inthe CG image. For example, when the same color as the chroma key coloris desired to be used in the CG image, a color obtained by changing onebit of the chroma key color may be used.

The reprojection unit 84 of the head-mounted display 100 converts thecamera image for which the image signal processing and the distortioncorrection processing have been executed in conformity to the latestposition of the point of view and the latest direction of the line ofsight and supplies the resulting image to the distortion processing unit86. The distortion processing unit 86 executes the distortion processingfor the camera image after the reprojection. The AR superimposing unit88 generates an augmented reality image by superimposing the CG chromakey image supplied from the image generating apparatus 200 on the cameraimage after the distortion processing. The generated augmented realityimage is displayed on the display panel 32.

In the above-described image generating system according to the premisetechnique, the CG chroma key image generated by the chroma keygenerating unit 235 undergoes the post-process by the post-process unit236, the reprojection processing by the reprojection unit 240, and thedistortion processing by the distortion processing unit 242. Therefore,aliasing occurs at the boundary of the virtual object and a false colorthat does not exist actually occurs, and unnaturalness becomesconspicuous when the CG chroma key image is superimposed on the cameraimage by the AR superimposing unit 88. Furthermore, the transmissioninterface of the general image is compatible with RGB but is notcompatible with RGBA including the alpha value. Therefore, it isdifficult to transmit the alpha value and thus the image generatingsystem is also subject to the limitation that it is difficult to expressa translucent CG image. To express a shadow cast on a real space by avirtual object and a reflection of the virtual object onto the realspace, the translucent CG image is desired to be synthesized with acamera image.

Image generating systems according to several embodiments that overcomethe problems in the image generating system according to the premisetechnique will be described below. However, description overlapping withthe premise technique is omitted as appropriate and configurationsimproved from the premise technique will be described.

A first embodiment will be described. The configuration of thehead-mounted display 100 is the same as that illustrated in FIG. 7.

FIG. 10 is a functional configuration diagram of the image generatingapparatus 200 according to the first embodiment.

The HDMI transmitting-receiving unit 280 receives a camera imagephotographed by the camera unit 80 and depth information from thehead-mounted display 100 and supplies the camera image to a camera imageacquiring unit 252 and supplies the depth information to the depthacquiring unit 250. This depth information is the depth information ofthe camera image of a real space and is referred to as “camera depthinformation.”

The image generating unit 230 reads out data for generation of computergraphics from the image storing unit 260 and carries out rendering ofobjects of a virtual space to generate a CG image. Then, the imagegenerating unit 230 superimposes the CG image on the camera imageprovided from the camera image acquiring unit 252 to generate aprovisional superposition image. In addition, the image generating unit230 generates a chroma key image from the CG image based on the cameradepth information provided from the depth acquiring unit 250. Thepost-process, the reprojection processing, and the distortion processingare executed for the provisional superposition image and thereprojection processing and the distortion processing are executed forthe chroma key image. Note that the post-process is not executed for thechroma key image. At last, a final synthesized CG chroma key image isgenerated by masking the provisional superposition image by the chromakey image and is output to the image storing unit 260.

The image generating unit 230 includes the rendering unit 232, asuperimposing unit 234, the chroma key generating unit 235, thepost-process unit 236, reprojection units 240 a and 240 b, distortionprocessing units 242 a and 242 b, and a synthesizing unit 244.

The rendering unit 232 generates the CG image through rendering ofobjects of a virtual space that are visible in the direction of the lineof sight from the position of the point of view of the user who wearsthe head-mounted display 100 in accordance with the position of thepoint of view and the direction of the line of sight of the user set bythe point-of-view/line-of-sight setting unit 220. When the renderingunit 232 carries out the rendering of the objects of the virtual space,the depth information of the virtual objects (referred to as “scenedepth information”) is written to a depth buffer for virtual spacerendering (referred to as “scene depth buffer”) and the front-rearpositional relation between the virtual objects is determined. As forpixels at which a virtual object is not rendered, a specific depth valueis not written in the scene depth buffer and the scene depth value isinfinite (indefinite).

Moreover, the rendering unit 232 carries out rendering of real objectsof a real space photographed by the camera unit 80. Shape informationand depth information of a thing of the real world are obtained bymaking a 3D scan of the space of the real world and carrying out spatialrecognition. For example, it is possible to acquire the depthinformation of the real space by using a depth sensor of a system of aninfrared pattern, structured light, time of flight (TOF), or the like oracquire the depth information of the real space from parallaxinformation of a stereo camera. As above, the real space is subjected tothe 3D scan in advance and modeling thereof is carried out with apolygon mesh structure. Rendering of wall, floor, ceiling, still things,and so forth of the real space is carried out by the rendering unit 232.However, they are rendered with only white without setting colorinformation. When the rendering unit 232 carries out the rendering ofthe objects of the real space, the depth information of the real objects(referred to as “real space depth information”) is written to a depthbuffer for real space rendering (referred to as “real space depthbuffer”) and the front-rear positional relation between the real objectsis determined.

The reason why the real space depth buffer is set separately from thescene depth buffer is because, if the depth value is written to thescene depth buffer in rendering of the real space, it becomes difficultto discriminate the scene depth value and the real space depth value andit becomes difficult to determine whether or not to specify each regionas the chroma key region.

The rendering unit 232 renders, as a translucent CG image, expressionrelating to light of a virtual space with respect to a real space,specifically shadows cast on real objects by virtual objects andreflections of the virtual objects onto the real space, expression inwhich the background of an object of the virtual space existing on thenear side is visible in a see-through manner, lighting expression basedon a virtual light source in the virtual space, and so forth. Forexample, shadow mapping can render shadows and reflections by using amethod in which a map of the depth from a light source is projected ontoa plane or a technique such as ray tracing. By superimposing thetranslucent CG image of shadows and reflections of virtual objects on alow-resolution camera image of a real space, the shadows and reflectionsof the virtual objects with respect to the real space can be expressed.Objects of the real space are rendered with only white and therefore canbe discriminated from the region in which the shadow or reflection isrendered. The rendering region of the real space is specified as thechroma key region whereas the region in which the shadow or reflectionis rendered is not specified as the chroma key region.

The rendering unit 232 gives the CG image in which the virtual objectsand the expression relating to light of the virtual space, such asshadows and reflections of the virtual objects, are rendered to thesuperimposing unit 234 and the chroma key generating unit 235.

The superimposing unit 234 superimposes the CG image on the camera imagewith low resolution and delay given from the camera image acquiring unit252 to generate a provisional superposition image and give it to thepost-process unit 236. The superimposing unit 234 superimposes thecamera image with low resolution and delay on the region that is theregion in which the scene depth value is infinite (i.e. region in whicha virtual object is not rendered) and is the region for which the realspace depth value is written. Due to this, the camera image with lowresolution and delay is superimposed on the CG image in which shadowsand reflections are rendered in a translucent manner, with the colorinformation of the shadows and reflections left. That is, the colorinformation of the shadows and reflections and the color information ofthe camera image with low resolution and delay are alpha-blended.

Here, note that the camera image is not superimposed on the region forwhich the real space depth value is not written and the scene depthvalue is written. If the mesh structure of a real space is deformed andan empty region such as a hole is set as described later, the real spacedepth value is not written for the empty region and the scene depthvalue is written. Therefore, the color of the empty region does notbecome the chroma key color and thus the camera image is notsuperimposed on the empty region. It also becomes possible to, instead,carry out rendering of a virtual space in the empty region and give aneffect by which the far side is visible from the empty region.

The post-process unit 236 executes the post-process for the provisionalsuperposition image and executes post-processing in such a manner thatthe provisional superposition image looks natural and smooth.

The first reprojection unit 240 a receives the latestposition-orientation information of the head-mounted display 100 fromthe position-orientation acquiring unit 210 and executes thereprojection processing for the provisional superposition image forwhich the post-process has been executed to convert the image to animage viewed from the latest position of the point of view and thelatest direction of the line of sight of the head-mounted display 100.

The first distortion processing unit 242 a executes the distortionprocessing for the provisional superposition image for which thereprojection processing has been executed and gives the resulting imageto the synthesizing unit 244.

The chroma key generating unit 235 generates a chroma key image based onthe camera depth information given from the depth acquiring unit 250 andthe scene depth information and the real space depth information givenfrom the rendering unit 232. Specifically, the chroma key generatingunit 235 generates the chroma key image in which the region that is theregion in which the scene depth value is infinite or the real spacedepth value is written and is other than the region in which a shadow orreflection is rendered is painted out with the chroma key color and thepart of the real object existing on the near side relative to thevirtual object is painted out with the chroma key color throughreference to the camera depth information and the scene depthinformation.

The post-process is not executed for the chroma key image.

The second reprojection unit 240 b receives the latestposition-orientation information of the head-mounted display 100 fromthe position-orientation acquiring unit 210 and executes thereprojection processing for the chroma key image to convert the image toan image viewed from the latest position of the point of view and thelatest direction of the line of sight of the head-mounted display 100.

The second distortion processing unit 242 b executes the distortionprocessing for the chroma key image for which the reprojectionprocessing has been executed and gives the resulting image to thesynthesizing unit 244.

The synthesizing unit 244 carries out synthesis with use of the chromakey image as a mask for the provisional superposition image to generatea synthesized CG chroma key image and store it in the image storing unit260.

Here, when the first reprojection unit 240 a executes the reprojectionprocessing for the provisional superposition image and the firstdistortion processing unit 242 a executes the distortion processing forthe provisional superposition image after the reprojection processing,generally interpolation processing such as bilinear interpolation isexecuted at the time of sampling of pixels. On the other hand, when thesecond reprojection unit 240 b executes the reprojection processing forthe chroma key image and the second distortion processing unit 242 bexecutes the distortion processing for the chroma key image after thereprojection processing, point sampling is carried out at the time ofsampling of pixels. This is because, if interpolation processing such asbilinear interpolation is executed for the chroma key image, the chromakey color is mixed with a non-chroma-key color and becomes another colorand the meaning of the chroma key image is lost.

In the reprojection processing and the distortion processing for thechroma key image, the following two methods may be used besides theabove-described point sampling. In the first method, bilinearinterpolation is carried out between the chroma key color andnon-chroma-key colors and pixel values of intermediate colors are oncecalculated at the time of sampling. However, a predetermined thresholdis set. Furthermore, if the pixel value after the interpolation is equalto or smaller than the threshold, the pixel value is set to the chromakey color. If the pixel value after the interpolation exceeds thethreshold, the pixel value is set to the original non-chroma-key color.According to this method, the intermediate colors generated due to theinterpolation are discarded and whether the color becomes the chroma keycolor or becomes the original non-chroma-key color is determineddepending on whether the pixel value exceeds the threshold. Therefore,the chroma key color is not mixed with the non-chroma-key color.

In the second method, when the chroma key image is generated, samplingof neighboring four pixels of a point to which (u, v) refers is carriedout. Furthermore, the color is set to the chroma key color if all of theneighboring four pixels apply to the condition of the chroma key and thecolor is not set to the chroma key color if all of the neighboring fourpixels do not apply to the condition of the chroma key. As the way ofselecting the neighboring pixels, selecting nine pixels may be employedbesides selecting four pixels. According to this method, the edgeboundary part is not painted out with the chroma key color. Thus, the CGimage and the camera image look to merge more naturally near theboundary.

The HDMI transmitting-receiving unit 280 reads out frame data of thesynthesized CG chroma key image generated by the image generating unit230 from the image storing unit 260 and transmits the frame data to thehead-mounted display 100 in accordance with the HDMI.

FIG. 11 is a diagram explaining the configuration of an image generatingsystem according to the first embodiment for superimposing a CG image ona camera image to generate an augmented reality image.

Camera image and camera depth information of the external worldphotographed by the camera unit 80 of the head-mounted display 100 aresupplied to the image signal processing unit 82. The image signalprocessing unit 82 executes the image signal processing and thedistortion correction processing for the camera image with low delay andhigh resolution and gives the resulting image to the reprojection unit84. Moreover, the image signal processing unit 82 transmits the cameraimage for which the image signal processing and the distortioncorrection processing have been executed and the camera depthinformation to the image generating apparatus 200. The camera image issupplied to the superimposing unit 234 and the camera depth informationis supplied to the chroma key generating unit 235. The camera imagetransmitted to the image generating apparatus 200 involves delay and hasa low resolution.

The rendering unit 232 of the image generating apparatus 200 carries outrendering of real objects of a real space. In addition, the renderingunit 232 generates virtual objects viewed from the position of the pointof view and the direction of the line of sight of the user who wears thehead-mounted display 100 and carries out rendering of expressionrelating to light of the virtual space with respect to the real space,specifically shadows cast on real objects by virtual objects andreflections of virtual objects, making virtual objects translucent,lighting expression based on a virtual light source, and so forth. Therendering unit 232 gives the generated CG image to the superimposingunit 234 and gives the scene depth information and the real space depthinformation to the chroma key generating unit 235.

The superimposing unit 234 superimposes the camera image on the CG imageto generate a provisional superposition image and give it to thepost-process unit 236. Here, the camera image provided from thehead-mounted display 100 may have a low resolution and involve delay.This is because the part of the camera image in the provisionalsuperposition image is masked by the chroma key image to be finallyerased. The superimposing unit 234 superimposes the camera image on theregion for which the scene depth value is not written and the real spacedepth value is written. Therefore, the camera image is superimposed onthe region in which the virtual object is not rendered and the region inwhich the expression relating to the light of the virtual space such asa shadow or reflection is rendered. In the region in which theexpression relating to the light of the virtual space such as a shadowor reflection is rendered, the color information of the expressionrelating to the light of the virtual space such as a shadow orreflection is synthesized with the low-resolution camera image as atranslucent CG image.

The post-process unit 236 executes the post-process for the provisionalsuperposition image. The reprojection unit 240 a converts theprovisional superposition image for which the post-process has beenexecuted in conformity to the latest position of the point of view andthe latest direction of the line of sight. The distortion processingunit 242 a executes the distortion processing on the provisionalsuperposition image after the reprojection and gives the resulting imageto the synthesizing unit 244.

The chroma key generating unit 235 generates a chroma key image based onthe camera depth information, the scene depth information, and the realspace depth information. The region for which the scene depth value isnot written is painted out with the chroma key color. However, thetranslucent CG region in which the expression relating to the light ofthe virtual space such as a shadow or reflection is rendered is notpainted out with the chroma key color. Furthermore, the positionalrelation between the objects of the real space and the objects of thevirtual space is determined in real time based on the camera depthinformation and the scene depth information, and the part of the realmoving thing (hand, ball, or the like) existing on the near siderelative to the virtual object is painted out with the chroma key color.The reprojection unit 240 b converts the chroma key image in conformityto the latest position of the point of view and the latest direction ofthe line of sight. The distortion processing unit 242 b executes thedistortion processing for the chroma key image after the reprojectionand gives the resulting image to the synthesizing unit 244.

The synthesizing unit 244 generates a synthesized CG chroma key image bymasking the provisional superposition image by the chroma key image.Here, the provisional superposition image has been subjected to thepost-process and therefore is a smooth image. Therefore, the boundarybetween the camera image and the CG image is not conspicuous. On theother hand, the post-process has not been executed for the chroma keyimage. Thus, aliasing and false color do not occur at the boundaries ofthe virtual objects. Therefore, when the superposition image for whichthe post-process has been executed is masked by the chroma key image forwhich the post-process has not been executed, the synthesized CG chromakey image that does not involve aliasing and false color at theboundaries of the virtual objects and is natural and smooth issynthesized.

Although the scene depth value is infinite, the translucent CG region inwhich a shadow or reflection is rendered is not painted out with thechroma key color. Therefore, the translucent CG region of the shadow orreflection is left in the synthesized CG chroma key image in the stateof being superimposed on the low-resolution camera image.

The synthesized CG chroma key image is transmitted to the head-mounteddisplay 100 as an RGB image in which specific one color is specified forthe chroma key, and is supplied to the AR superimposing unit 88.

The reprojection unit 84 of the head-mounted display 100 converts thecamera image with low delay and high resolution for which the imagesignal processing and the distortion correction processing have beenexecuted in conformity to the latest position of the point of view andthe latest direction of the line of sight and supplies the resultingimage to the distortion processing unit 86. The distortion processingunit 86 executes the distortion processing for the camera image with lowdelay and high resolution after the reprojection. The AR superimposingunit 88 generates an augmented reality image by superimposing thesynthesized CG chroma key image supplied from the image generatingapparatus 200 on the camera image with low delay and high resolutionafter the distortion processing. The generated augmented reality imageis displayed on the display panel 32. The camera image with low delayand high resolution is superimposed on the region for which the chromakey color is specified on the side of the head-mounted display 100.

FIG. 12 is a diagram explaining an augmented reality image obtained bysuperimposing a CG image on a camera image by the image generatingsystem according to the first embodiment. Similarly to FIG. 5, a userattempts to touch the teapot 410 that is a virtual object and the teapot410, which is the virtual object, is superimposed on the camera image inwhich the hand 420 is captured. A shadow 412 cast on the real table bythe teapot 410, which is the virtual object, is rendered as translucentCG in the form of being superimposed on the camera image. Thus, thefloral pattern on the surface of the real table is visible in asee-through manner in the rendering region of the shadow 412.

FIG. 13 is a diagram explaining a synthesized CG chroma key image usedby the image generating system according to the first embodiment. In thesynthesized CG chroma key image, the rendering region of a virtualobject (here, teapot) and the rendering region of a shadow or reflectionof the virtual object (here, shadow cast on the table surface with afloral pattern by the teapot) are CG and the other region is painted outwith the chroma key color. Furthermore, if a real moving thing (here,hand of the user) exists on the near side relative to the virtualobject, the part of the moving thing on the near side is also paintedout with the chroma key color.

The synthesized CG chroma key image of FIG. 13 is superimposed on ahigh-resolution camera image on the side of the head-mounted display 100and thereby the augmented reality image of FIG. 12 is generated. Notethat, although the part of the shadow of the teapot 410 is synthesizedwith a low-resolution camera image, the camera image other than thispart has a high resolution including the region of the hand of the user.

When the image generating system of the first embodiment is used, it isalso possible to virtually make a hole in a wall, ceiling, table, or thelike of a real space and superimpose a CG image thereon. This will bedescribed with reference to FIG. 14 and FIG. 15.

FIG. 14 is a diagram explaining an example in which a polygon mesh of areal space is deformed to make a hole in a wall. The polygon mesh of thereal space is obtained by a 3D scan and a hole 430 can be set in thewall as in FIG. 14 by deforming the polygon mesh. As another expressionmethod of the hole 430, a texture with an alpha mask may be used.

No thing of the real space exists in the hole 430. Therefore, when therendering unit 232 carries out rendering of the real space, the depthinformation is not written to the region corresponding to the hole 430in the real space depth buffer. Meanwhile, the rendering unit 232 cancarry out rendering of a virtual object in the hole 430 and, on thisoccasion, the depth information is written to the region correspondingto the hole 430 in the scene depth buffer. Therefore, when thesuperimposing unit 234 superimposes a camera image on a CG image, thecamera image is not superimposed on the hole 430, for which the realspace depth value is not set, and the CG image is displayed in the hole430, for which the scene depth value is set.

FIG. 15 is a diagram explaining an example in which a virtual object isrendered in the hole of the wall of the real space. In FIG. 15, avirtual hole 440 is rendered based on CG corresponding to the hole 430in FIG. 14 and a virtual car is rendered based on CG on the far side ofthe virtual hole 440.

According to the image generating system of the first embodiment, asense of discomfort regarding the boundary between the camera image andthe CG image is alleviated by the post-process. In addition, thepost-process is not executed for the chroma key image. Therefore, thechroma key image without aliasing and false color can be generated andthe synthesized CG chroma key image with high quality can be generated.Because this synthesized CG chroma key image is superimposed on thecamera image to generate the augmented reality image, the augmentedreality image without unnaturalness can be generated. Furthermore, alsowhen a translucent part exists in the CG image, the translucent part canbe superimposed on the camera image and translucency processing can beexecuted when the post-processing is executed. Thus, the translucentpart can also be expressed although latency due to the rendering exists.Moreover, the synthesized CG chroma key image is an RGB image in whichspecific one color is specified as the chroma key image and thus can betransmitted by a general communication interface, such as the HDMI, thatcan transmit the RGB image.

In the above description, the synthesized CG chroma key image is an RGBimage in which specific one color is specified as the chroma key image.However, the chroma key color of the chroma key image may be stored inthe alpha component of RGBA. Specifically, until intermediate processingof the post-process, processing may be executed for the part set to thechroma key color in the image, with the alpha component of RGBA madetransmissive. Then, the color of the part in which the alpha componentis transmissive may be replaced by the chroma key color at the finalstage of the post-process. This eliminates the need to use two differentframe buffers for storing the provisional superposition image and thechroma key image and processing of the provisional superposition imageand the chroma key image can be executed on one frame buffer.

A second embodiment will be described. The configuration of thehead-mounted display 100 is basically the same as that illustrated inFIG. 7. However, the reprojection unit 84 has a first reprojection unit84 a for a camera image and a second reprojection unit 84 b for asynthesized CG chroma key image.

FIG. 16 is a functional configuration diagram of the image generatingapparatus 200 according to the second embodiment.

The HDMI transmitting-receiving unit 280 receives a camera imagephotographed by the camera unit 80 and depth information from thehead-mounted display 100 and supplies the camera image to the cameraimage acquiring unit 252 and supplies the depth information to the depthacquiring unit 250.

The image generating unit 230 reads out data for generation of computergraphics from the image storing unit 260 and carries out rendering ofobjects of a virtual space to generate a CG image. Then, the imagegenerating unit 230 superimposes the CG image on the camera imageprovided from the camera image acquiring unit 252 to generate aprovisional superposition image. In addition, the image generating unit230 generates a chroma key image from the CG image based on the cameradepth information provided from the depth acquiring unit 250. Thepost-process is executed for the provisional superposition image whereasthe post-process is not executed for the chroma key image. At last, afinal synthesized CG chroma key image is generated by masking theprovisional superposition image by the chroma key image and is output tothe image storing unit 260.

The image generating unit 230 includes the rendering unit 232, thesuperimposing unit 234, the chroma key generating unit 235, thepost-process unit 236, and the synthesizing unit 244.

The rendering unit 232 generates the CG image through rendering ofobjects of a virtual space that are visible in the direction of the lineof sight from the position of the point of view of the user who wearsthe head-mounted display 100 in accordance with the position of thepoint of view and the direction of the line of sight of the user set bythe point-of-view/line-of-sight setting unit 220. When the renderingunit 232 carries out the rendering of the objects of the virtual space,the scene depth information is written to the scene depth buffer.

Moreover, the rendering unit 232 carries out rendering of real objectsof a real space photographed by the camera unit 80. When the renderingunit 232 carries out the rendering of the objects of the real space, thereal space depth information of the real objects is written to the realspace depth buffer.

The rendering unit 232 gives the CG image in which the virtual objectsand expression relating to light of the virtual space, such as shadowsand reflections of the virtual objects, are rendered to thesuperimposing unit 234 and the chroma key generating unit 235.

The superimposing unit 234 superimposes the CG image on the camera imagewith low resolution and delay given from the camera image acquiring unit252 to generate a provisional superposition image and give it to thepost-process unit 236. The superimposing unit 234 superimposes thecamera image with low resolution and delay on the region that is theregion in which the scene depth value is infinite and is the region forwhich the real space depth value is written.

The post-process unit 236 executes the post-process for the provisionalsuperposition image and executes post-processing in such a manner thatthe provisional superposition image looks natural and smooth.

The chroma key generating unit 235 generates a chroma key image based onthe camera depth information given from the depth acquiring unit 250 andthe scene depth information and the real space depth information givenfrom the rendering unit 232.

The synthesizing unit 244 carries out synthesis with use of the chromakey image as a mask for the provisional superposition image to generatea synthesized CG chroma key image and store it in the image storing unit260.

The HDMI transmitting-receiving unit 280 reads out frame data of thesynthesized CG chroma key image generated by the image generating unit230 from the image storing unit 260 and transmits the frame data to thehead-mounted display 100 in accordance with the HDMI.

FIG. 17 is a diagram explaining the configuration of an image generatingsystem according to the second embodiment for superimposing a CG imageon a camera image to generate an augmented reality image.

Camera image and camera depth information of the external worldphotographed by the camera unit 80 of the head-mounted display 100 aresupplied to the image signal processing unit 82. The image signalprocessing unit 82 executes the image signal processing and thedistortion correction processing for the camera image with low delay andhigh resolution and gives the resulting image to the reprojection unit84. Moreover, the image signal processing unit 82 transmits the cameraimage for which the image signal processing and the distortioncorrection processing have been executed and the camera depthinformation to the image generating apparatus 200. The camera image issupplied to the superimposing unit 234 and the camera depth informationis supplied to the chroma key generating unit 235. The camera imagetransmitted to the image generating apparatus 200 involves delay and hasa low resolution.

The rendering unit 232 of the image generating apparatus 200 carries outrendering of real objects of a real space. In addition, the renderingunit 232 generates virtual objects viewed from the position of the pointof view and the direction of the line of sight of the user who wears thehead-mounted display 100 and carries out rendering of lightingexpression such as shadows cast on real objects by virtual objects andreflections of virtual objects. The rendering unit 232 gives thegenerated CG image to the superimposing unit 234 and gives the scenedepth information and the real space depth information to the chroma keygenerating unit 235.

The superimposing unit 234 superimposes the camera image on the CG imageto generate a provisional superposition image and give it to thepost-process unit 236. Similarly to the first embodiment, the cameraimage provided from the head-mounted display 100 may have a lowresolution and involve delay.

The post-process unit 236 executes the post-process for the provisionalsuperposition image and gives the resulting image to the synthesizingunit 244.

The chroma key generating unit 235 generates a chroma key image based onthe camera depth information, the scene depth information, and the realspace depth information and gives it to the synthesizing unit 244.

The synthesizing unit 244 generates a synthesized CG chroma key image bymasking the superposition image by the chroma key image. Similarly tothe first embodiment, when the superposition image for which thepost-process has been executed is masked by the chroma key image forwhich the post-process has not been executed, the synthesized CG chromakey image that does not involve aliasing and false color at theboundaries of the virtual objects and is natural and smooth issynthesized.

The synthesized CG chroma key image is transmitted to the head-mounteddisplay 100 as an RGB image in which specific one color is specified forthe chroma key, and is supplied to the reprojection unit 84 b.

The first reprojection unit 84 a of the head-mounted display 100converts the camera image with low delay and high resolution for whichthe image signal processing and the distortion correction processinghave been executed in conformity to the latest position of the point ofview and the latest direction of the line of sight and supplies theresulting image to the AR superimposing unit 88.

The second reprojection unit 84 b of the head-mounted display 100converts the synthesized CG chroma key image in conformity to the latestposition of the point of view and the latest direction of the line ofsight and supplies the resulting image to the AR superimposing unit 88.

Here, the reason why the reprojection unit is divided into the firstreprojection unit 84 a for a camera image and the second reprojectionunit 84 b for a synthesized CG chroma key image in the head-mounteddisplay 100 is because a long time is taken for rendering of the imagegenerating apparatus 200 and the amount of difference that should becorrected differs depending on reprojection. For example, reprojectionof the next frame is executed in the first reprojection unit 84 awhereas reprojection of the frame after the next frame is desired to beexecuted in the second reprojection unit 84 b.

The AR superimposing unit 88 generates an augmented reality image bysuperimposing the synthesized CG chroma key image for which thereprojection processing has been executed by the second reprojectionunit 84 b on the camera image with low delay and high resolution forwhich the reprojection processing has been executed by the firstreprojection unit 84 a, and supplies the augmented reality image to thedistortion processing unit 86. The camera image with low delay and highresolution is superimposed on the region for which the chroma key coloris specified on the side of the head-mounted display 100.

The distortion processing unit 86 executes the distortion processing forthe augmented reality image. The generated augmented reality image isdisplayed on the display panel 32.

According to the image generating system of the second embodiment,similarly to the first embodiment, the synthesized CG chroma key imageis superimposed on the camera image to generate the augmented realityimage. Therefore, there is an advantage that the augmented reality imagewithout unnaturalness can be generated. Besides, the following advantageexists. Differently from the first embodiment, the reprojectionprocessing is executed for the camera image and the synthesized CGchroma key image on the side of the head-mounted display 100. Thus, thecamera image and the synthesized CG chroma key image can be converted inconformity to the position of the point of view and the direction of theline of sight immediately before display on the display panel 32 and theaugmented reality image with tracking capability can be provided withhigh accuracy. Furthermore, the burden of the reprojection processing onthe side of the image generating apparatus 200 can be alleviated. Thus,more resources can be applied to rendering on the side of the imagegenerating apparatus 200.

In the first embodiment and the second embodiment, the configurations inwhich the post-process by the post-process unit 236 is not executed forthe chroma key image are described. However, as a modification example,a configuration in which adjustment of the point of view and apost-process for scaling are applied to the chroma key image may beemployed. In this case, if a method in which the average of surroundingpixels is used, or the like, is employed when pixels are interpolated,the color of the boundary changes to a different color from the chromakey color. Therefore, the post-process is desired to be applied in sucha manner that the chroma key image is not changed. Alternatively, aftera normal post-process with which the color of the boundary changes isapplied to the chroma key image, only the region that completely matchesthe original chroma key color may be used as a mask.

The present disclosure is described above based on the embodiments. Theembodiments are exemplification and it will be understood by thoseskilled in the art that various modification examples are possible incombinations of the respective constituent elements and the respectiveprocessing processes of them and that such modification examples alsofall within the range of the present disclosure.

In the above description, kinds of processing such as adjustment of thedepth of field, tone mapping, and anti-aliasing are exemplified as thepost-process. However, kinds of processing including distortionprocessing, simple scaling, trapezoidal transform, and so forth may alsobe referred to as the post-process.

The present disclosure contains subject matter related to that disclosedin Japanese Priority Patent Application JP 2018-195220 filed in theJapan Patent Office on Oct. 16, 2018 and Japanese Priority PatentApplication JP 2019-079475 filed in the Japan Patent Office on Apr. 18,2019, the entire contents of which are hereby incorporated by reference.

What is claimed is:
 1. An image generating apparatus comprising: arendering unit configured to carry out rendering of an object of avirtual space and an object of a real space and carry out rendering ofexpression relating to light of the virtual space with respect to thereal space to generate a computer graphics image; a superimposing unitconfigured to superimpose the computer graphics image on a photographedimage of the real space to generate a provisional superposition image; achroma key generating unit configured to generate a chroma key imagethrough executing chroma key processing for the computer graphics imagebased on depth information of the photographed image of the real space;and a synthesizing unit configured to generate a synthesized chroma keyimage used for being superimposed on the photographed image of the realspace to generate an augmented reality image by applying a mask to theprovisional superposition image by the chroma key image, wherein thechroma key generating unit employs a region of the real space in whichthe object of the virtual space is not rendered as a chroma key regionand does not employ a region of the real space in which the expressionrelating to the light of the virtual space exists as a chroma keyregion.
 2. The image generating apparatus according to claim 1, whereinthe expression relating to the light of the virtual space with respectto the real space is at least one of a shadow or reflection of theobject of the virtual space onto the object of the real space,expression in which a background of the object of the virtual space isvisible in a see-through manner, and lighting expression based on avirtual light source in the virtual space.
 3. The image generatingapparatus according to claim 1, further comprising: deforming a polygonmesh structure obtained by carrying out spatial recognition of the realspace to generate an empty region in which rendering of a virtual spaceis carried out, wherein the rendering unit carries out the rendering ofthe object of the real space based on the deformed polygon meshstructure.
 4. The image generating apparatus according to claim 1,further comprising: a post-process unit configured to execute apost-process for the provisional superposition image, wherein thesynthesizing unit generates the synthesized chroma key image by applyinga mask to the provisional superposition image for which a post-processhas been executed by the chroma key image for which a post-process hasnot been executed.
 5. The image generating apparatus according to claim1, further comprising: a reprojection unit configured to convert theprovisional superposition image for which a post-process has beenexecuted and the chroma key image in conformity to a new position of apoint of view or a new direction of a line of sight, wherein thesynthesizing unit generates the synthesized chroma key image by applyinga mask to the provisional superposition image for which reprojectionprocessing has been executed by the chroma key image for whichreprojection processing has been executed.
 6. The image generatingapparatus according to claim 1, wherein the photographed image of thereal space used when the superimposing unit generates the provisionalsuperposition image has a lower resolution than the photographed imageof the real space used for generating the augmented reality image. 7.The image generating apparatus according to claim 6, wherein theexpression relating to the light of the virtual space with respect tothe real space is superimposed on the photographed image with the lowerresolution as a translucent computer graphics image.
 8. An imagegenerating system comprising: a head-mounted display; and an imagegenerating apparatus, wherein the image generating apparatus includes arendering unit configured to carry out rendering of an object of avirtual space and an object of a real space and carry out rendering ofexpression relating to light of the virtual space with respect to thereal space to generate a computer graphics image, a first superimposingunit configured to superimpose the computer graphics image on aphotographed image of the real space transmitted from the head-mounteddisplay to generate a provisional superposition image, a chroma keygenerating unit configured to generate a chroma key image throughexecuting chroma key processing for the computer graphics image based ondepth information of the photographed image of the real spacetransmitted from the head-mounted display, and a synthesizing unitconfigured to generate a synthesized chroma key image used for beingsuperimposed on the photographed image of the real space to generate anaugmented reality image by applying a mask to the provisionalsuperposition image by the chroma key image, the head-mounted displayincludes a second superimposing unit configured to generate theaugmented reality image by synthesizing the photographed image of thereal space with the synthesized chroma key image transmitted from theimage generating apparatus, and the chroma key generating unit employs aregion of the real space in which the object of the virtual space is notrendered as a chroma key region and does not employ a region of the realspace in which the expression relating to the light of the virtual spaceexists as a chroma key region.
 9. An image generating method comprising:carrying out rendering of an object of a virtual space and an object ofa real space and carrying out rendering of expression relating to lightof the virtual space with respect to the real space to generate acomputer graphics image; superimposing the computer graphics image on aphotographed image of the real space to generate a provisionalsuperposition image; generating a chroma key image through executingchroma key processing for the computer graphics image based on depthinformation of the photographed image of the real space; and generatinga synthesized chroma key image used for being superimposed on thephotographed image of the real space to generate an augmented realityimage by applying a mask to the provisional superposition image by thechroma key image, wherein the generating a chroma key image employs aregion of the real space in which the object of the virtual space is notrendered as a chroma key region and does not employ a region of the realspace in which the expression relating to the light of the virtual spaceexists as a chroma key region.
 10. A non-transitory, computer readablestorage medium containing a program, which when executed by a computer,causes the computer to perform an image generating method by carryingout actions, comprising: carrying out rendering of an object of avirtual space and an object of a real space and carrying out renderingof expression relating to light of the virtual space with respect to thereal space to generate a computer graphics image; superimposing thecomputer graphics image on a photographed image of the real space togenerate a provisional superposition image; generating a chroma keyimage through executing chroma key processing for the computer graphicsimage based on depth information of the photographed image of the realspace; and generating a synthesized chroma key image used for beingsuperimposed on the photographed image of the real space to generate anaugmented reality image by applying a mask to the provisionalsuperposition image by the chroma key image, wherein the generating achroma key image employs a region of the real space in which the objectof the virtual space is not rendered as a chroma key region and does notemploy a region of the real space in which the expression relating tothe light of the virtual space exists as a chroma key region.